Structured Co-reference Graph Attention for Video-grounded Dialogue

A video-grounded dialogue system referred to as the Structured Co-reference Graph Attention (SCGA) is presented for decoding answer sequence a question regarding given video while keeping track of context. Although recent efforts have made great strides in improving quality response, performance still far from satisfactory. The two main challenging issues are follows: (1) how deduce co-reference among multiple modalities and (2) reason on rich underlying semantic structure with complex spatial temporal dynamics. To this end, SCGA based Resolver that performs dereferencing via building structured graph over modalities, Spatio-temporal Video Reasoner captures local-to-global dynamics gradually neighboring attention. makes use pointer network dynamically replicate parts sequence. validity proposed demonstrated AVSD@DSTC7 AVSD@DSTC8 datasets, benchmarks, TVQA dataset, large-scale videoQA benchmark. Our empirical results show outperforms other state-of-the-art systems both extensive ablation study qualitative analysis reveal gain improved interpretability.